Stereo matching method and apparatus

ABSTRACT

A stereo matching method includes extracting feature points of a first image and feature points of a second image, the first image and the second image together constituting a stereo image, determining reference points by matching the feature points of the second image to the feature points of the first image, classifying the reference points, and performing stereo matching on pixels of which disparities are not determined in the first image and the second image based on disparities of the reference points in the pixels determined based on a result of the classifying.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit under 35 USC § 119(a) of KoreanPatent Application No. 10-2017-0012360 filed on Jan. 26, 2017, in theKorean Intellectual Property Office, the entire disclosure of which isincorporated herein by reference for all purposes.

BACKGROUND 1. Field

The following description relates to a stereo matching method andapparatus.

2. Description of Related Art

A human can determine distances by comparing two images acquired atdifferent locations. Stereo matching refers to an automatedcomputer-based visual field technique for determining distances much inthe way a human can by comparing two images acquired at differentlocations. Stereo matching utilizes a left image and a right image. Theleft image and the right image may be aligned to achieve an appropriatestereo effect. Corresponding pairs may be extracted from an alignedstereo image through stereo matching. Disparities of the correspondingpairs may be used to obtain depth information.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, a stereo matching method includes extractingfeature points of a first image and feature points of a second image,the first image and the second image together constituting a stereoimage, determining reference points by matching the feature points ofthe second image to the feature points of the first image, classifyingthe reference points and performing stereo matching on pixels of whichdisparities are not determined in the first image and the second imagebased on disparities of the reference points in the pixels determinedbased on a result of the classifying.

The determining of the reference points may include matching the featurepoints of the second image to the feature points of the first imageusing a window-based correlation, and determining, as the referencepoints, feature points having an optimal cost among the matched featurepoints using a cost analysis for measuring the window-based correlation.

The classifying of the reference points may include classifying thereference points into a class based on at least one of whether thereference points are present in a region in which a depth discontinuityoccurs, whether the reference points are present in a region in which anocclusion occurs, whether the reference points are present in a regionhaving a texture value less than or equal to a preset reference, andwhether the reference points have disparity values greater than or equalto a preset reliability.

The classifying of the reference points may include classifying thereference points into a class based on at least one of whether aconsistency is maintained by the matching, a ratio between a firstoptimal cost and a second optimal cost among correlation costs of thereference points calculated based on a cost analysis for measuring awindow-based correlation, and whether optimal costs are detected amongthe correlation costs of the reference points.

The classifying of the reference points further may include classifyinga reference point for which the consistency is not maintained by thematching among the reference points into a first class, classifying areference point for which the consistency is maintained and the ratiobetween the first optimal cost and the second optimal cost is less thana preset threshold among the reference points into a second class, andclassifying a reference point for which the consistency is maintainedand the ratio between the first optimal cost and the second optimal costreaches the preset threshold among the reference points into a thirdclass.

The classifying of the reference points may further include labeling thereference points based on the respective classes of the referencepoints.

The performing of stereo matching may include performing stereo matchingon the pixels by updating disparity values of the reference points basedon a reliability of the disparities of the reference points.

The performing of stereo matching on the pixels by updating thedisparity values further may include, based on the reliability of thedisparities of the reference points, propagating a disparity value of areference point classified into a third class to a disparity value of areference point classified into at least one of a first class or asecond class, the reference point classified into the third class beingadjacent to the reference point classified into at least one of thefirst class or the second class among the reference points, andperforming stereo matching on the pixels based on the propagateddisparity value.

The performing of stereo matching on the pixels based on the propagateddisparity value may further include determining a polygon that uses thereference points respectively classified into the first class, thesecond class, and the third class as vertices, determining a searchrange for calculating disparities of pixels present in the polygon basedon the propagated disparity value, and performing stereo matching on thepixels based on the search range.

The performing of stereo matching on the pixels by updating thedisparity values of the reference points may include resetting a windowthat extracts the disparity values of the reference points classifiedinto the first class and the second class from each of the first imageand the second image and performing stereo matching on the pixels basedon the reset window.

The resetting of the window may include resetting the window based on atleast one of a shifted window having an adjustable angle and multiplewindows having identical sizes with respect to a reference point ofwhich a depth is discontinued from adjacent reference points among thereference points classified into the first class and the second class,and resetting the window based on an extension window obtained byextending a size of the window with respect to a reference point ofwhich a depth is continued from the adjacent reference points among thereference points classified into the first class and the second class,based on a result of the classifying.

A non-transitory computer-readable storage medium may store instructionsthat, when executed by a processor, cause the processor to perform themethod described above.

A stereo matching apparatus includes a processor and a memory configuredto store a computer-readable instruction where, in response to executionof the instruction at the processor, the processor is configured toextract feature points of a first image and feature points of a secondimage, the first image and the second image together constituting astereo image, determine reference points by matching the feature pointsof the second image to the feature points of the first image, classifythe reference points, determine disparities of the reference pointsbased on a result of the classifying, and perform stereo matching onpixels of which disparities are not determined in the first image andthe second image based on disparities of the reference points determinedbased on a result of the classifying.

The processor may be further configured to match the feature points ofthe second image to the feature points of the first image using awindow-based correlation, and determine, as the reference points,feature points having an optimal cost among the matched feature pointsusing a cost analysis for measuring the window-based correlation, andthe memory is configured to store information including disparity valuescorresponding to the reference points.

The processor is may be further configured to classify the referencepoints into a class based on at least one of whether the referencepoints are present in a region in which a depth discontinuity occurs,whether the reference points are present in a region in which anocclusion occurs, whether the reference points are present in a regionhaving a texture value less than or equal to a preset reference, andwhether the reference points have disparity values greater than or equalto a preset reliability.

The processor may be further configured to classify the reference pointsinto a class based on at least one of whether a consistency ismaintained by the matching, a ratio between a first optimal cost and asecond optimal cost among correlation costs of the reference pointscalculated based on a cost analysis for measuring a window-basedcorrelation, and whether optimal costs are detected among thecorrelation costs of the reference points.

The processor may be further configured to classify a reference pointfor which the consistency is not maintained by the matching among thereference points into a first class, classify a reference point forwhich the consistency is maintained and the ratio between the firstoptimal cost and the second optimal cost is less than a preset thresholdamong the reference points into a second class, and classify a referencepoint for which the consistency is maintained and the ratio between thefirst optimal cost and the second optimal cost reaches the presetthreshold among the reference points into a third class.

The processor may be further configured to perform stereo matching onthe pixels by updating disparity values of the reference points based ona reliability of the disparities of the reference points.

The processor may be further configured to, based on the reliability ofthe disparities of the reference points, propagate a disparity value ofa reference point classified into a third class to a disparity value ofa reference point classified into at least one of a first class or asecond class, the reference point classified into the third class beingadjacent to the reference point classified into at least one of thefirst class or the second class among the reference points, and performstereo matching on the pixels based on the propagated disparity value.

The processor may be further configured to determine a polygon that usesthe reference points respectively classified into the first class, thesecond class, and the third class as vertices, determine a search rangefor calculating disparities of pixels in the polygon based on thepropagated disparity value, and perform stereo matching on the pixelsbased on the search range.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a stereo matching process.

FIG. 2A and 2B illustrate examples of reference points used for stereomatching.

FIG. 3 is a flowchart illustrating an example of a stereo matchingmethod.

FIG. 4 is a flowchart illustrating an example of a method of determiningreference points.

FIG. 5 is a flowchart illustrating an example of a method of classifyingreference points.

FIG. 6 is a flowchart illustrating an example of a method of performingstereo matching on pixels of which disparities are not determined in animage.

FIG. 7 is a flowchart illustrating another example of a stereo matchingmethod.

FIG. 8 is a block diagram illustrating an example of a stereo matchingapparatus.

Throughout the drawings and the detailed description, the same referencenumerals refer to the same elements. The drawings may not be to scale,and the relative size, proportions, and depiction of elements in thedrawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known in the art may be omitted forincreased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

Terms such as first, second, A, B, (a), (b), and the like may be usedherein to describe components. Each of these terminologies is not usedto define an essence, order or sequence of a corresponding component butused merely to distinguish the corresponding component from othercomponent(s). For example, a first component may be referred to a secondcomponent, and similarly the second component may also be referred to asthe first component.

The terminology used herein is for describing various examples only, andis not to be used to limit the disclosure. The articles “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. The terms “comprises,” “includes,”and “has” specify the presence of stated features, numbers, operations,members, elements, and/or combinations thereof, but do not preclude thepresence or addition of one or more other features, numbers, operations,members, elements, and/or combinations thereof.

Unless otherwise defined, all terms including technical and scientificterms used herein have the same meaning as commonly understood by one ofordinary skill in the art to which this disclosure pertains. Terms, suchas those defined in commonly used dictionaries, are to be interpreted ashaving a meaning that is consistent with their meaning in the context ofthe relevant art, and are not to be interpreted in an idealized oroverly formal sense unless expressly so defined herein.

Hereinafter, reference will now be made in detail to examples withreference to the accompanying drawings, wherein like reference numeralsrefer to like elements throughout.

FIG. 1 illustrates an example of a stereo matching process. Referring toFIG. 1, a stereo matching apparatus 110 receives a stereo image andoutputs disparities of pixels included in the stereo image. The stereomatching apparatus 110 may be configured using at least one softwaremodule, at least one hardware module, or various combinations thereof.

The stereo image includes a left image L and a right image R. The stereomatching apparatus 110 detects pixels corresponding to pixels includedin the left image L from the right image R, or detects pixelscorresponding to pixels included in the right image R from the leftimage L. Although it is described that corresponding pixels are detectedfrom the right image R based on the left image L below, otherconfigurations are contemplated. For example, corresponding pixels maybe also detected from the left image L based on the right image R. Inaddition, one of the left image L and the right image R may be referredto as a first image, and the other one may be referred to as a secondimage.

The stereo matching apparatus 110 detects corresponding pixel pairs fromthe left image L and the right image R, and calculates disparities ofthe corresponding pixel pairs. Depth information of an object includedin the stereo image may be determined based on the disparities. Thedepth information may be used to render a three-dimensional (3D) imageor to measure a distance between a user and the object. For example, anactual object and a virtual object may be mixed in an output image tocreate an image in augmented reality (AR). Here, a virtual object may besmoothly disposed between actual objects based on depth information ofthe actual objects.

The depth information may be variously used by a small device, forexample, a mobile device and the like. In the case of the small device,battery capacity or computing resources are limited. Accordingly, anaccuracy of stereo matching may be maintained and the amount of time andresources used for stereo matching may be reduced and minimized.

In detail, the stereo matching apparatus 110 may sample reference pixelsalong contour lines included in a first image, and perform edgedetection by recognizing pixels and edges adjacent to a boundary showinga sudden depth variation in the contour lines. For example, as shown inFIG. 1, the edges are provided in a shape of a straight line, a polygon,a curved line, a circle, or a point.

The stereo matching apparatus 110 may sample the reference pixels fromfeature points included in various features such as edges, corners, andpoints, for example, a meeting point of different objects and/or apattern created by texture. The stereo matching apparatus 110 may usethe sampled reference pixels to generate a polygonal mesh. Under asmoothness constraint that groups similar depth values in one object,pixels having similar depths may be included in a predetermined polygon.Thus, a depth variation may be estimated based on the polygonal mesh,and the stereo matching apparatus 110 may effectively perform stereomatching on the pixels using the polygonal mesh.

FIG. 1 illustrates reference pixels including a reference pixel 10 inthe left image L.

The stereo matching apparatus 110 detects corresponding pixels ofreference pixels from a second image through stereo matching, andcalculates disparities of the reference pixels based on locations of thereference pixels and locations of the corresponding pixels. For example,the stereo matching apparatus 110 may scan a region corresponding to themaximum search range from the right image R using a window of apredetermined size. When the left image L and the right image R arehorizontally aligned, a y coordinate of the scanned region may bedetermined from the right image R based on a y coordinate of thereference pixel 10 of the left image L. In response to detecting acorresponding pixel 50 from the right image R, the stereo matchingapparatus 110 may calculate a disparity of the reference pixel 10 basedon a difference between an x coordinate of the reference pixel 10 and anx coordinate of the corresponding pixel 50.

The stereo matching apparatus 110 determines a polygon that uses thereference pixels as vertices in the first image, and estimates (ordetermines) the search range including disparities of pixels presentwithin the polygon based on the disparities of the reference pixels. Theleft image L includes triangles that use reference pixels as vertices.The triangles may form a triangular mesh. Although the left image Lshows the triangular mesh only on a partial region, the triangular meshmay be formed over the entire region of an image. Examples using thetriangular mesh are described herein. Also, a rectangular mesh or apentagonal mesh may be formed using the reference pixels. Alternatively,a mesh in various types of polygons, for example, a triangle, arectangle, a pentagon, and the like, may be formed using the referencepixels.

An edge included in an image may represent a boundary between an objectand a background and/or a boundary between objects. Thus, depthinformation may significantly vary along the edge used as the boundary.Depth information may vary at a predictable level in a portion excludingthe edge. For example, a discontinuity may be present between adisparity of a pixel present within a triangle 20 and a disparity of apixel present within a triangle 30.

The stereo matching apparatus 110 classifies reference pixels which arepresent at a boundary and are not matched to each other in the leftimage L and the right image R, and performs stereo matching on pixels ofeach image based on the disparities of the reference pixels determinedbased on a result of classifying.

The stereo matching apparatus 110 classifies the reference pixels basedon, for example, a visual cue of an image and a feature of an estimateddisparity value.

The stereo matching apparatus 110 classifies the reference points basedon at least one of whether the reference points are present in a regionin which a depth discontinuity occurs, a region in which an occlusionoccurs, a region having a texture value less than or equal to a presetreference, and/or whether the reference points have a disparity valuegreater than or equal to a preset reliability.

The stereo matching apparatus 110 may enhance an accuracy of stereomatching by varying a method of determining (estimating) a disparity anda range for searching for a disparity of a related pixel based on aclass of each reference point. Detailed description of a reference pointused for stereo matching is provided with reference to FIG. 2.

The stereo matching apparatus 110 estimates a search range based on areference value that varies depending on pixel groups. For example, asearch range for performing stereo matching on a pixel present withinthe triangle 30 is calculated based on disparities of three referencepixels that constitute the triangle 30. A search range for performingstereo matching on a pixel present within the triangle 20 may becalculated based on disparities of three reference pixels thatconstitute the triangle 20.

Here, disparities of reference pixels present on a line segment 40 maycorrespond to an object, instead of corresponding to a background. Inthis case, a search range estimated for performing stereo matching onthe pixel present within the triangle 30 may not include an actualdisparity of the corresponding pixel present within the triangle 20. Thestereo matching apparatus 110 may adjust the disparities of thereference pixels present on the line segment 40 such that the searchrange for performing stereo matching on the pixel present within thetriangle 20 includes the actual disparity of the corresponding pixel.

FIGS. 2A and 2B illustrate examples of reference points used for stereomatching. Referring to FIGS. 2A and 2B, SP1, SP2, SP3, and SP4correspond to reference points that are robustly matched in a right eyeimage and a left eye image, and SP5 corresponds to an occluded referencepoint present at a boundary in any one of the right eye image and theleft eye image. The reference points may be determined by matchingfeature points of the right eye image and feature points of the left eyeimage.

FIG. 2A illustrates an example in which stereo matching is performedusing reference points that are robustly matched in a right eye imageand a left eye image.

An edge or a boundary portion of an object is accurately detected fromeach image. However, when the edge or the boundary portion is occludedby any one of the right eye image and the left eye image, the stereomatching may be unavailable. Thus, a pixel corresponding to the edge orthe boundary portion may be excluded from the reference points used forstereo matching as illustrated in FIG. 2A. In this case, a disparity maybe inaccurately determined because stereo matching is performed onlyusing a mesh and the matched pixels of the right eye image and the lefteye image irrespective of boundary information of an object.

In FIG. 2A, a disparity of a pixel P_(i) may be found in a search rangedetermined based on a triangle including reference points SP1, SP2, andSP3 as vertices. The search range of the disparity of the pixel P_(i)may be set based on a value of disparity interpolation between referencepoints SP1 and SP3. Here, because reference point SP3 is present inanother object, for example, a bowling pin, which is different from anobject, for example, a bowling ball, in which the pixel P_(i) ispresent, invalid information on the disparity of the pixel P_(i) may beprovided. Thus, an error in a calculation result of a disparity mayincrease.

A disparity of a pixel P_(i) may be found in a search range determinedbased on a triangle including reference points SP2, SP3, and SP4 asvertices. Here, the search range of the disparity of the pixel P_(i) maybe set based on a value of disparity interpolation between referencepoints SP2 and SP3. Because reference point SP3 is present in the objectwhich is different from the object, for example, a bowling ball, inwhich the pixel P_(i) is present, invalid information on the disparityof the pixel P_(j) may be provided. Thus, an error in a calculationresult of a disparity may increase.

FIG. 2B illustrates an example in which stereo matching is performedbased on reference points present at a boundary. Here, the referencepoints are not matched in a right eye image and a left eye image.

When SP5 present at the boundary is given as a reference point, thedisparity of the pixel P_(i) may be found in a search range determinedbased on a triangle including reference points SP1, SP2, and SP5 asvertices. The search range of the disparity of the pixel P_(i) may beset based on a value of the disparity interpolation between referencepoints SP1 and SP5. Because reference point SP5 is present in theobject, for example, a bowling ball, in which the pixel P_(i) ispresent, valid information on the disparity of the pixel Pi may beprovided. Thus, an accurate calculation result may be obtained.

In addition, even when the triangle including the pixel P_(i) does notperfectly fit into the boundary of the object, SP5 may correct andreduce an error caused by using reference point SP3 that is present in adifferent object.

Because a light intensity is drastically changed in a portion, forexample, at a boundary of an object, in which a depth continuity occurs,a plurality of reference points may be detected. However, referencepoints present at the boundary may be occluded so that the referencepoints may be present only in one of the right image and the left image.The occluded reference points may be excluded from a set of referencepoints through a consistency check that checks whether the referencepoints are consistently shown (represented) in the right eye image andthe left eye image for robust matching.

However, the reference points at the boundary (or edge) in each imagemay well represent the corresponding image such that the referencepoints should be reflected to constitute each triangle when a mesh isgenerated through triangulation. Thus, a disparity may be accuratelycalculated by considering a discontinuity in a following pixel matchingprocess.

Accordingly, in an example, all reference points may be classified intoclasses and may be included in sets of reference points in lieu ofexcluding the occluded reference points that are present at the boundary(or edge). Thus, more accurate disparity information and a more accurateresult of stereo matching may be obtained.

However, in a case of reference points that are not able to be matchedin the right eye image and the left eye image, disparity estimationthrough matching may be unavailable. In an example, based on asmoothness constraint condition in which a disparity value is minimal inan identical object, the stereo matching is performed by settingdisparity values of unmatched reference points as adjacent reliablereference points, that is, disparity values of reference points matchedin the right eye image and the left eye image.

In an example, even when ambiguous reference points are detected from alow texture region, the disparity values of the adjacent reliablereference points may be used for pixel matching by setting the disparityvalues of the adjacent reliable reference points as disparity values ofthe ambiguous reference points.

FIG. 3 is a flowchart illustrating an example of a stereo matchingmethod. Referring to FIG. 3, in operation 310, a stereo matchingapparatus extracts feature points of a first image and feature points ofa second image. The first image and second image together constitute astereo image. The first image may be any one of a left image and a rightimage.

To extract the feature points of the first image and the feature pointsof the second image, the stereo matching apparatus may use, for example,a Canny operator for edge detection, a Sobel operator, a Laplacian ofGaussian (LoG) operator, a difference of Gaussian (DoG) operator, aHarris operator for corner detection, a Census transform operator forrepresenting a regional space, a Prewitt operator, a Roberts operator, aLaplacian operator, and/or a Compass operator.

In response to the rectified first image and the rectified second imagebeing input, the stereo matching apparatus may extract the featurepoints based on various pieces of information including, for example, alight intensity and/or a relative order of the light intensity,obtainable from each image.

In operation 320, the stereo matching apparatus determines the referencepoints by matching the feature points of the second image to the featurepoints of the first image. Here, “matching” is also referred to as acorrespondence between the feature points of the first image and thefeature points of the second image.

The stereo matching apparatus may determine a reference point set usingthe reference points. The stereo matching apparatus may check whether aright eye image and a left eye image are matched in anappropriately-defined full disparity range set for application withrespect to the reference point set. For example, when a general baselinewithin a threshold length is defined, the full disparity range may beset to be a half size of a scan line of an image. Disparity values ofthe reference points may be calculated in response to matching thereference points in operation 320. A detailed description of a method bywhich the stereo matching apparatus determines the reference points isprovided with reference to FIG. 4.

In operation 330, the stereo matching apparatus classifies the referencepoints determined in operation 320. In response to matching thereference points in operation 320, the stereo matching apparatusclassifies the reference points into classes based on at least one ofwhether the reference points are present in a region in which a depthdiscontinuity occurs, a region in which an occlusion occurs, and aregion having a texture value less than or equal to a preset reference,and/or whether the reference points have the disparity values greaterthan or equal to a preset reliability.

The stereo matching apparatus classifies the reference points intoclasses based on at least one of whether a consistency is maintained bythe matching, a ratio between a first optimal cost and a second optimalcost among correlation costs of the reference points calculated based ona cost analysis for measuring a window-based correlation, and whether aplurality of optimal costs are detected. Here, the “optimal cost” may beunderstood as referring to a minimum cost. The stereo matching apparatusmay label the reference points based on the class of each classifiedreference point. Detailed description of a method by which the stereomatching apparatus classifies the reference points into classes isprovided with reference to FIG. 5.

In operation 340, the stereo matching apparatus performs stereo matchingon pixels of which disparities are not determined in the first image andthe second image based on disparities of the reference points based on aresult of the classifying in operation 330. That is, the stereo matchingapparatus may perform stereo matching on pixels of which disparityvalues are not calculated (or determined) in an image based on thedisparities of the reference points determined based on the result ofthe classifying in operation 330.

In operation 340, the stereo matching apparatus generates atwo-dimensional (2D) mesh corresponding to the first image.Alternatively, the stereo matching apparatus may generate a 2D meshcorresponding to the first image and a 2D mesh corresponding to thesecond image to cross-check whether a consistency of the referencepoints is maintained by the matching. The stereo matching apparatus mayperform stereo matching on the pixels of which disparities are notdetermined in an image by updating the disparity values of the referencepoints from the 2D mesh based on a reliability of the disparities of thereference points determined based on the result of the classifying inoperation 330. A detailed description of a method by which the stereomatching apparatus performs stereo matching on the pixels of whichdisparities are not determined in the image is provided with referenceto FIG. 6.

In an example, the stereo matching apparatus resets a window thatextracts the disparity values of the reference points from the firstimage and the second image based on the result of classifying, andperforms stereo matching on the pixels based on the reset window.

FIG. 4 is a flowchart illustrating an example of a method of determiningreference points. Referring to FIG. 4, in operation 410, the stereomatching apparatus matches the feature points of the second image to thefeature points of the first image based on a window-based correlation.The stereo matching apparatus may perform a correspondence check betweenthe feature points of the first image and the feature points of thesecond image based on the window-based correlation that is generallyused for local stereo matching.

In operation 420, the stereo matching apparatus determines, as referencepoints, feature points having an optimal cost among the feature pointsmatched in operation 410 based on a cost analysis for measuring thewindow-based correlation. Here, the optimal cost may be determined by aminimum value or a maximum value based on a definition of the costanalysis. For example, a sum of absolute differences (SAD), a sum ofsquared differences (SSD), and a normalized cross correlation (NCC) areused as a cost analysis for measuring a correlation.

In operation 430, the stereo matching apparatus stores informationincluding the disparity values corresponding to the reference points.The stereo matching apparatus may store additional informationincluding, for example, a ratio between a first optimal cost and asecond optimal cost, and whether a plurality of optimal cost values arepresent in order to use the additional information for classifying thereference points.

FIG. 5 is a flowchart illustrating an example of a method of classifyingreference points. Referring to FIG. 5, in operation 510, the stereomatching apparatus verifies whether reference point consistency ismaintained by matching. In operation 520, the stereo matching apparatusclassifies a reference point for which the consistency is not maintainedby the matching among the reference points into a first class. Here, thereference point for which the consistency is not maintained by thematching may correspond to an occluded reference point in any one of aright eye image and a left eye image.

In operation 530, the stereo matching apparatus verifies whether a ratiobetween a first optimal cost and a second optimal cost among correlationcosts with respect to the reference point for which the consistency ismaintained is less than a preset threshold. Here, the ratio between thefirst optimal cost and the second optimal cost among the correlationcosts being less than the preset threshold may indicate that adifference between the first optimal cost and the second optimal cost isunclear. The reference point for which the difference between the firstoptimal cost and the second optimal cost is unclear may correspond to anambiguous reference point having an insufficient texture value.

In operation 540, the stereo matching apparatus classifies a referencepoint for which the ratio between the first optimal cost and the secondoptimal cost is less than the preset threshold into a second class.

In operation 550, the stereo matching apparatus classifies a referencepoint for which the consistency is maintained and the ratio between thefirst optimal cost and the second optimal cost reaches the presetthreshold into a third class. Here, the reference point classified intothe third class may correspond to a stable reference point having areliable disparity value.

FIG. 6 is a flowchart illustrating an example of a method of performingstereo matching on pixels of which disparities are not determined in animage. Referring to FIG. 6, in operation 610, the stereo matchingapparatus generates a two-dimensional (2D) mesh corresponding to a firstimage. The stereo matching apparatus may generate the 2D mesh on aprovided image using reference points. For example, the stereo matchingapparatus may generate the 2D mesh based on Delaunay triangulation thatconstitutes a triangular network by connecting a set of points intriangles.

In operation 620, the stereo matching apparatus propagates a disparityvalue of a reference point classified into a third class to a disparityvalue of a reference point classified into at least one of a first classor a second class among reference points of the 2D mesh generated inoperation 610 based on a reliability of disparities of the referencepoints. Here, the reference point classified into the third class isadjacent to the reference point classified into the first class or thesecond class.

For example, in a case of the reference point classified into the firstclass or the second class, a disparity value obtained through stereomatching may be unreliable. Thus, after triangulation is performed, thestereo matching apparatus may propagate an adjacent reference pointhaving a reliable disparity value in a triangle including the referencepoint classified into the first class or the second class, that is, thedisparity value of the reference point classified into the third class,such that the reference points may be used as the disparity value of thereference point classified into the first class or the second class.

In operation 630, the stereo matching apparatus determines a polygon,for example, a triangle, that uses the reference points as vertices.Here, the reference points may be classified into the first class or thesecond class, in addition to the third class. The reference pointsclassified into the third class may have reliable disparity values bymatching a right eye image and a left eye image. Thus, when the polygonis constituted, the reference points classified into the third class maybe determined as vertices of the polygon. Also, the reference pointsclassified into the first class or the second class may be determined asthe vertices of the polygon. In operation 640, the stereo matchingapparatus determines a search range for calculating disparities ofpixels in the polygon based on the propagated disparity value. Thestereo matching apparatus may determine the disparities by narrowing thesearch range based on the disparity values of the reference pointsconstituting the triangle when the disparity values of the pixelsincluded in the polygon, for example, triangle, are determined.

In operation 650, the stereo matching apparatus performs stereo matchingon the pixels of which disparities are not determined in the image basedon the search range determined in operation 640.

FIG. 7 is a flowchart illustrating another example of a stereo matchingmethod. Referring to FIG. 7, in operation 705, the stereo matchingapparatus receives a first image and a second image. The first image andthe second image may correspond to a right eye image and a left eyeimage.

In operation 710, the stereo matching apparatus extracts feature pointsof the first image and feature points of the second image. In operation715, the stereo matching apparatus determines reference points bymatching the feature points of the second image to the feature points ofthe first image.

In operation 720, the stereo matching apparatus classifies the referencepoints determined in operation 715.

In operation 725, the stereo matching apparatus verifies whether a depthof a reference point is discontinued from adjacent reference pointsamong the reference points classified into a first class and a secondclass based on a result of the classifying in operation 720.

Based on a result of the verifying that the depth of the reference pointis discontinued from the adjacent reference points in operation 725, thestereo matching apparatus resets a window based on at least one of ashifted window having an adjustable angle or multiple windows having anidentical size in operation 730. In operation 710, the stereo matchingapparatus extracts the feature points of the first image and the featurepoints of the second image based on the window reset in operation 730.

Based on the result of the verifying that the depth of the referencepoint is not discontinued from the adjacent reference points inoperation 725, the stereo matching apparatus performs matching on pixelsby updating disparity values of the reference points based on areliability of the disparities of the reference points determined basedon a result of the classifying, in operation 745.

In operation 735, the stereo matching apparatus verifies whether thedepth of the reference point is continued from the adjacent referencepoints among the reference points classified into the first class andthe second class based on a result of the classifying in operation 720.

Based on a result of the verifying that the depth of the reference pointis continued from the adjacent reference points among the referencepoints classified into the first class and the second class in operation735, the stereo matching apparatus resets the window based on anextension window obtained by extending a size of the window with respectto the reference point of which the depth is continued from the adjacentreference points in operation 740. In operation 710, the stereo matchingapparatus extracts the feature points of the first image and the featurepoints of the second image based on the window reset in operation 740.

Based on the result of the verifying that the depth of the referencepoint is not continued from the adjacent reference points among thereference points classified into the first class and the second class inoperation 735, the stereo matching apparatus performs stereo matching onpixels of which disparities are not determined in an image by updatingthe disparity values of the reference points based on the reliability ofthe disparities of the reference points determined based on the resultof the classifying, in operation 745.

FIG. 8 is a block diagram illustrating an example of a stereo matchingapparatus. Referring to FIG. 8, a stereo matching apparatus 800 includesa processor 810, a sensor 820, and a memory 830. The processor 810, thesensor 820, and the memory 830 may communicate with each other via acommunication bus 840.

The processor 810 may process the above-described operations associatedwith stereo matching. In more detail, the processor 810 extracts featurepoints of a first image and feature points of a second image, anddetermines reference points by matching the feature points of the secondimage to the feature points of the first image. The processor 810classifies the reference points and performs stereo matching on pixelsof which disparities are not determined in the first image and thesecond image based on disparities of the reference points determinedbased on a result of the classifying.

The processor 810 matches the feature points of the second image to thefeature points of the first image based on a window-based correlation,and determines, as the reference points, feature points having anoptimal cost among the feature points matched based on a cost analysisfor measuring the window-based correlation. In addition, the processor810 may perform at least one of the above-described methods of FIGS. 1through 7, and a further description is omitted. The processor 810executes instructions or programs, or controls the stereo matchingapparatus 800.

The sensor 820 may capture a stereo image. The sensor 820 may include afirst sensor configured to capture a left image and a second sensorconfigured to capture a right image. Each of the first sensor and thesecond sensor may be, for example, an image sensor, a proximity sensor,or an infrared ray sensor. The sensor 820 may capture a stereo image,using, for example, a method of converting an optical image into anelectrical signal. The sensor 820 may transfer at least one of acaptured color image, a captured depth image, and a captured infraredray image to the at least one of the processor 810 and/or the memory830.

The memory 830 stores information including disparity valuescorresponding to the reference points. Also, the memory 830 may storedata associated with the aforementioned stereo matching. For example,the memory 830 may store a stereo image, information on the extractedfeature points, information on the determined reference points,information on a result of the classifying of the reference points, anda result of stereo matching.

The memory 830 stores computer-readable instructions. In response toexecution of instructions stored in the memory 830 at the processor 810,the processor 810 may perform an operation associated with theaforementioned stereo matching.

The stereo matching apparatus 800 may be connected to an externaldevice, for example, a personal computer (PC) and a network (not shown),through an input/output device, and may exchange data. The stereomatching apparatus 800 may be configured as at least a portion of amobile device, for example, a mobile phone, a smartphone, a personaldigital assistant (PDA), a tablet computer, a laptop computer, etc., acomputing device, for example, a PC, a tablet computer, a net-book,etc., or an electronic device, for example, a television, a smarttelevision, a security device for controlling a gate, etc. Theaforementioned description may be applicable to the stereo matchingapparatus 800 and thus, a further description is omitted.

Examples of hardware components include controllers, sensors,generators, drivers, and any other electronic components known to one ofordinary skill in the art. In one example, the hardware components areimplemented by one or more processors or computers. A processor orcomputer is implemented by one or more processing elements, such as anarray of logic gates, a controller and an arithmetic logic unit, adigital signal processor, a microcomputer, a programmable logiccontroller, a field-programmable gate array, a programmable logic array,a microprocessor, or any other device or combination of devices known toone of ordinary skill in the art that is capable of responding to andexecuting instructions in a defined manner to achieve a desired result.In one example, a processor or computer includes, or is connected to,one or more memories storing instructions or software that are executedby the processor or computer. Hardware components implemented by aprocessor or computer execute instructions or software, such as anoperating system (OS) and one or more software applications that run onthe OS, to perform the operations described herein. The hardwarecomponents also access, manipulate, process, create, and store data inresponse to execution of the instructions or software. For simplicity,the singular term “processor” or “computer” may be used in thedescription of the examples described herein, but in other examplesmultiple processors or computers are used, or a processor or computerincludes multiple processing elements, or multiple types of processingelements, or both. In one example, a hardware component includesmultiple processors, and in another example, a hardware componentincludes a processor and a controller. A hardware component has any oneor more of different processing configurations, examples of whichinclude a single processor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

Instructions or software to control a processor or computer to implementthe hardware components and perform the methods as described above arewritten as computer programs, code segments, instructions or anycombination thereof, for individually or collectively instructing orconfiguring the processor or computer to operate as a machine orspecial-purpose computer to perform the operations performed by thehardware components and the methods as described above. In one example,the instructions or software include machine code that is directlyexecuted by the processor or computer, such as machine code produced bya compiler. In another example, the instructions or software includehigher-level code that is executed by the processor or computer using aninterpreter. Programmers of ordinary skill in the art can readily writethe instructions or software based on the block diagrams and the flowcharts illustrated in the drawings and the corresponding descriptions inthe specification, which disclose algorithms for performing theoperations performed by the hardware components and the methods asdescribed above.

The instructions or software to control a processor or computer toimplement the hardware components and perform the methods as describedabove, and any associated data, data files, and data structures, arerecorded, stored, or fixed in or on one or more non-transitorycomputer-readable storage media. Examples of a non-transitorycomputer-readable storage medium include read-only memory (ROM),random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs,CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs,BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-opticaldata storage devices, optical data storage devices, hard disks,solid-state disks, and any device known to one of ordinary skill in theart that is capable of storing the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and providing the instructions or software and any associateddata, data files, and data structures to a processor or computer so thatthe processor or computer can execute the instructions. In one example,the instructions or software and any associated data, data files, anddata structures are distributed over network-coupled computer systems sothat the instructions and software and any associated data, data files,and data structures are stored, accessed, and executed in a distributedfashion by the processor or computer.

While this disclosure includes specific examples, it will be apparent toone of ordinary skill in the art that various changes in form anddetails may be made in these examples without departing from the spiritand scope of the claims and their equivalents. The examples describedherein are to be considered in a descriptive sense only, and not forpurposes of limitation. Descriptions of features or aspects in eachexample are to be considered as being applicable to similar features oraspects in other examples. Suitable results may be achieved if thedescribed techniques are performed in a different order, and/or ifcomponents in a described system, architecture, device, or circuit arecombined in a different manner, and/or replaced or supplemented by othercomponents or their equivalents. Therefore, the scope of the disclosureis defined not by the detailed description, but by the claims and theirequivalents, and all variations within the scope of the claims and theirequivalents are to be construed as being included in the disclosure.

What is claimed is:
 1. A stereo matching method, comprising: extractingfeature points of a first image and feature points of a second image,the first image and the second image together constituting a stereoimage; determining reference points by matching the feature points ofthe second image to the feature points of the first image; classifyingthe reference points; and performing stereo matching on pixels of whichdisparities are not determined in the first image and the second imagebased on disparities of the reference points in the pixels determinedbased on a result of the classifying.
 2. The stereo matching method ofclaim 1, wherein the determining of the reference points comprises:matching the feature points of the second image to the feature points ofthe first image using a window-based correlation; and determining, asthe reference points, feature points having an optimal cost among thematched feature points using a cost analysis for measuring thewindow-based correlation.
 3. The stereo matching method of claim 1,wherein the classifying of the reference points comprises classifyingthe reference points into a class based on at least one of: whether thereference points are present in a region in which a depth discontinuityoccurs, whether the reference points are present in a region in which anocclusion occurs, whether the reference points are present in a regionhaving a texture value less than or equal to a preset reference, andwhether the reference points have disparity values greater than or equalto a preset reliability.
 4. The stereo matching method of claim 1,wherein the classifying of the reference points comprises classifyingthe reference points into a class based on at least one of: whether aconsistency is maintained by the matching, a ratio between a firstoptimal cost and a second optimal cost among correlation costs of thereference points calculated based on a cost analysis for measuring awindow-based correlation, and whether optimal costs are detected amongthe correlation costs of the reference points.
 5. The stereo matchingmethod of claim 4, wherein the classifying of the reference pointsfurther comprises: classifying a reference point for which theconsistency is not maintained by the matching among the reference pointsinto a first class; classifying a reference point for which theconsistency is maintained and the ratio between the first optimal costand the second optimal cost is less than a preset threshold among thereference points into a second class; and classifying a reference pointfor which the consistency is maintained and the ratio between the firstoptimal cost and the second optimal cost reaches the preset thresholdamong the reference points into a third class.
 6. The stereo matchingmethod of claim 4, wherein the classifying of the reference pointsfurther comprises labeling the reference points based on the respectiveclasses of the reference points.
 7. The stereo matching method of claim1, wherein the performing of stereo matching comprises performing stereomatching on the pixels by updating disparity values of the referencepoints based on a reliability of the disparities of the referencepoints.
 8. The stereo matching method of claim 7, wherein the performingof stereo matching on the pixels by updating the disparity valuesfurther comprises: based on the reliability of the disparities of thereference points, propagating a disparity value of a reference pointclassified into a third class to a disparity value of a reference pointclassified into at least one of a first class or a second class, thereference point classified into the third class being adjacent to thereference point classified into at least one of the first class or thesecond class among the reference points; and performing stereo matchingon the pixels based on the propagated disparity value.
 9. The stereomatching method of claim 8, wherein the performing of stereo matching onthe pixels based on the propagated disparity value further comprises:determining a polygon that uses the reference points respectivelyclassified into the first class, the second class, and the third classas vertices; determining a search range for calculating disparities ofpixels present in the polygon based on the propagated disparity value;and performing stereo matching on the pixels based on the search range.10. The stereo matching method of claim 5, wherein the performing ofstereo matching on the pixels by updating the disparity values of thereference points comprises: resetting a window that extracts thedisparity values of the reference points classified into the first classand the second class from each of the first image and the second image;and performing stereo matching on the pixels based on the reset window.11. The stereo matching method of claim 10, wherein the resetting of thewindow comprises: resetting the window based on at least one of ashifted window having an adjustable angle and multiple windows havingidentical sizes with respect to a reference point of which a depth isdiscontinued from adjacent reference points among the reference pointsclassified into the first class and the second class; and resetting thewindow based on an extension window obtained by extending a size of thewindow with respect to a reference point of which a depth is continuedfrom the adjacent reference points among the reference points classifiedinto the first class and the second class, based on a result of theclassifying.
 12. A non-transitory computer-readable storage mediumstoring instructions that, when executed by a processor, cause theprocessor to perform the method of claim
 1. 13. A stereo matchingapparatus, comprising: a processor; and a memory configured to store acomputer-readable instruction, wherein, in response to execution of theinstruction at the processor, the processor is configured to: extractfeature points of a first image and feature points of a second image,the first image and the second image together constituting a stereoimage, determine reference points by matching the feature points of thesecond image to the feature points of the first image, classify thereference points, determine disparities of the reference points based ona result of the classifying, and perform stereo matching on pixels ofwhich disparities are not determined in the first image and the secondimage based on disparities of the reference points determined based on aresult of the classifying.
 14. The stereo matching apparatus of claim13, wherein the processor is further configured to: match the featurepoints of the second image to the feature points of the first imageusing a window-based correlation, and determine, as the referencepoints, feature points having an optimal cost among the matched featurepoints using a cost analysis for measuring the window-based correlation,and the memory is configured to store information including disparityvalues corresponding to the reference points.
 15. The stereo matchingapparatus of claim 13, wherein the processor is further configured toclassify the reference points into a class based on at least one of:whether the reference points are present in a region in which a depthdiscontinuity occurs, whether the reference points are present in aregion in which an occlusion occurs, whether the reference points arepresent in a region having a texture value less than or equal to apreset reference, and whether the reference points have disparity valuesgreater than or equal to a preset reliability.
 16. The stereo matchingapparatus of claim 13, wherein the processor is further configured to:classify the reference points into a class based on at least one of:whether a consistency is maintained by the matching, a ratio between afirst optimal cost and a second optimal cost among correlation costs ofthe reference points calculated based on a cost analysis for measuring awindow-based correlation, and whether optimal costs are detected amongthe correlation costs of the reference points.
 17. The stereo matchingapparatus of claim 16, wherein the processor is further configured to:classify a reference point for which the consistency is not maintainedby the matching among the reference points into a first class, classifya reference point for which the consistency is maintained and the ratiobetween the first optimal cost and the second optimal cost is less thana preset threshold among the reference points into a second class, andclassify a reference point for which the consistency is maintained andthe ratio between the first optimal cost and the second optimal costreaches the preset threshold among the reference points into a thirdclass.
 18. The stereo matching apparatus of claim 13, wherein theprocessor is further configured to perform stereo matching on the pixelsby updating disparity values of the reference points based on areliability of the disparities of the reference points.
 19. The stereomatching apparatus of claim 18, wherein the processor is furtherconfigured to, based on the reliability of the disparities of thereference points, propagate a disparity value of a reference pointclassified into a third class to a disparity value of a reference pointclassified into at least one of a first class or a second class, thereference point classified into the third class being adjacent to thereference point classified into at least one of the first class or thesecond class among the reference points, and perform stereo matching onthe pixels based on the propagated disparity value.
 20. The stereomatching apparatus of claim 18, wherein the processor is furtherconfigured to determine a polygon that uses the reference pointsrespectively classified into the first class, the second class, and thethird class as vertices, determine a search range for calculatingdisparities of pixels in the polygon based on the propagated disparityvalue, and perform stereo matching on the pixels based on the searchrange.